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Abstract: /t is essential to establish 

a multi-dimensional postgraduate 
quality evaluation system for student 
assessment and training. This study aimed 
to explore the construction of the multi- 
index and hierarchical comprehensive 
evaluation system for postgraduate 
training in science and engineering based 
on the Context, Input, Process, Product 
(CIPP) model using Analytic Hierarchy 
Process. It involved 756 postgraduates 

in physics and engineering who were 
randomly selected via the Internet. Data 
were collected from the questionnaire 
about postgraduates’ basic information. 
After collection, Factor Analysis was used 
to verify the rationality of the design of 
second-level and third-level indicators, 
and adjust the corresponding weights. 
On this basis, Cluster Analysis was used 

to classify the training quality of the 
postgraduates based on their scores on 
academic ability, basic quality, and social 
ability indicators. The results revealed 
that the index system includes 4 first-level 
indicators, 12 second-level indicators and 
36 third-level indicators, and different 
weights being assigned to the indicators 
according to their influence on the training 
quality of postgraduates in science and 
engineering. This study also provides 
some reference for the quality of science 
and engineering postgraduate training in 
Chinese universities by proposing relevant 
measures, which could be interesting also 
for international audience. 
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Introduction 


Postgraduate education shoulders the important mission of high-level 
talent training and innovation and creation, which is an essential cornerstone 
of China’s development and social progress (State Council of China, 2020). 
Postgraduate education plays a key role in cultivating talent with cutting-edge 
knowledge, innovative abilities, critical thinking, and high-level skills as part 
of the national innovation strategy (Commission on the Future of Graduate 
Education, 2015). Quality evaluation of postgraduate education is a compre- 
hensive examination for the postgraduates, which has a positive impact on 
the management mode of postgraduates, the construction of the supervisory 
teams, scientific research, and classroom teaching. Different countries have 
different requirements for the quality training of postgraduate education. 
For example, postgraduate education at the United States universities must 
be validated by two authoritative assessment organizations, the Council on 
Postsecondary Accreditation (COPA) of the United States and Federal Ministry 
of Education, Office of Qualifications and Institutional Assessment, before 
bachelor’s, master’s, and doctoral degrees can be awarded. Russia has a 
complete system for assessing the quality of postgraduate training in higher 
education. The Japanese government has enacted a series of decrees to as- 
sess postgraduate education, which effectively ensures the quality of higher 
education. The German postgraduate education system has always puta high 
value on quality, and it is very strict regarding degree awarding and teaching. 
Each university in Britain has formulated strict regulations on the awarding 
of degrees. Regardless of the differences in national conditions, modern 
postgraduate education in the world today values the quality of degree 
conferral and the evaluation of postgraduate training (Zhang & Guo, 2009). 
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In recent years, the number of people applying for postgraduate school in China has been increasing year 
by year. Approximately 510,000 more people applied for postgraduate school in 2020 than 2019, and 360,000 
more people applied in 2021 than 2020. While in 2022, 4.57 million people took the “Unified National Graduate 
Entrance Examination’, an increase of 800,000 over 2021 (Ministry of Education, 2020a). Growing demand for 
postgraduate studies, driven by social and cultural factors, has forced Chinese universities to rapidly expand 
postgraduate degrees in a short period of time, making it difficult for universities to accommodate both qual- 
ity and growth demands. But the school system has to develop students’ specific literacies, including science, 
technology and engineering literacy in the frame of STEM literacy (Cencelj et al., 2020). Furthermore, only about 
23% of Chinese students are successful after an entrance examination and interview (Ministry of Education, 
2019). The Chinese postgraduates in science and engineering are a group of high-level talent who bears the 
double mission of high-end talent supply and science and technology innovation, which is crucial for helping 
the development of new engineering. Therefore, it is necessary to reconsider the goals, models, and content 
of postgraduate education (Quality Assurance in Postgraduate Education, 2010). 

Since the beginning of 21st century, the development of politics, economy and social culture has exerted 
more and more influence on the higher education environment. The changing environment has, to some ex- 
tent, influenced attitudes, goals and strategies of higher education training. In order to ensure that the quality 
development of higher education is in line with student development and academic standards, United Nations 
Educational, Scientific and Cultural Organization (UNESCO, 2004) and International Network of Quality Assur- 
ance Agencies in Higher Education (INQAAHE, 2005) have begun to attach importance to the use of quality 
assurance evaluation methods. The evaluation process is no longer limited to administrators but is increasingly 
involving other practitioners and no longer a single factor evaluation involving only school policies or student 
performances. Zaki (2020) evaluated the teaching performance of teachers majoring in English teaching by 
methods of quantitative evaluation models and online questionnaires. It pointed out that Quality Assurance 
Bureau (QAB) plays a positive role in improving teaching quality. Okpa et al. (2020) used the Student Participation 
Quality Assurance Management Questionnaire (SPQAMQ) to evaluate the student participation in decision- 
making and management aspects of the higher education evaluation process, pointing out the importance 
and necessity of student participation in evaluation and management. It can be seen that the evaluation of 
student educational quality should be a multi-subject and multi-factor evaluation system. 

The main methods of quality evaluation are internal evaluation, which evaluates exams, defenses, and 
learning engagement, and external evaluation, which evaluates externally responsible institutions. Rosa et al. 
(2016) surveyed and analyzed the attitudes of faculty and general faculty in universities from the perspective 
of internal evaluation (Internal Quality Assurance) on the four aspects of quality evaluation: goals, culture, obe- 
dience, and consistency, and concluded that timely internal evaluation helps to promote faculty and general 
staff awareness and understanding of the teaching and learning process and to improve critical and reflective 
evaluation. Goran and Anna (2019) conducted external quality evaluation with the help of the Institutional 
Evaluation Program (IEP) and used the results of this evaluation as a basis for improvement of peer review. 
Both evaluation approaches emphasize not only the importance of quality assurance evaluation, but also the 
feedback and retrospective nature of evaluation and the significance of teaching quality development. 

Based on the development trend of evaluation with multiple factors and perspectives, some scholars 
have proposed multi-dimensional evaluation models. Izci et al. (2020) emphasized the consistency of evalua- 
tion indicators with the learning process and established an evaluation model including four indicators, that 
is, teacher’s perspective, task characteristics and details, authentic implementation, and authentic training. 
Markus and Philipp (2018) designed three least squares regression models for evaluating administrators, 
individual functions, and integrated perspectives to verify their validity by evaluating the goals, strategies, 
and methods of higher education in this school. These evaluation approaches have taken into account the 
evaluation from different stages of teaching and learning, but the evaluation elements and indicators are still 
relatively few. Moreover, quality evaluation is a continuous and changing process, and various indicators need 
to be constantly reviewed and considered, and less attention has been paid to the revision and calibration 
aspects in these evaluation models. 

For China, with the rapid development of domestic science and technology and the emergence of new 
fields and technologies, the demand of society for innovative and compound talents who can quickly adapt to 
the development of science and technology and industrial reform is increasing (Ye et al., 2021). The establish- 
ment of an enterprise-centered, market-oriented, in-depth integration of technology innovation system is one 
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of the trends of science and engineering postgraduate education in China. The postgraduate training mode 
in China has shifted from the traditional and single disciplinary teaching mode to the “dual-track” mode of 
integrating multiple educational resources. The educational administrators of various universities are combin- 
ing interdisciplinary and cross-disciplinary teaching contents and establishing and perfecting the “dual tutor” 
employment system. The “dual tutor system’ refers to the selection and recruitment of one supervisor each from 
the enrollment unit and the relevant practice cultivation unit of the university during the cultivation process 
of professional degree master students, and the guidance of the supervisor of the enrollment unit is the main 
and the practice base. The main responsibilities of the on-campus supervisor are to cultivate students’ learning 
ability and enhance their development space, while the off-campus supervisors’ aim is to guide students to 
improve their practical operation ability, increase their practical cognitive ability, and adapt to their future work. 
On the basis of the continuous deepening of the quality assurance evaluation system of higher education in 
China, in order to adapt to the diversified training strategies of Chinese universities, meet the policy orienta- 
tion of innovative resource sharing and complementary advantages, and improve the evaluation efficiency, 
the National Education Commission of China (NECC) puts forward the requirements of multi-factor, multi-angle 
and quantitative evaluation on the evaluation of the quality of science and engineering postgraduate training. 


CIPP Model 


CIPP is an evaluation model covering the early, middle, and later period of activities. It was first proposed 
by Stufflebeam, and has the characteristics of decision, guidance, and feedback (Stufflebeam & Shinkfield, 
1966). After CIPP model was proposed, Ohara and Pickcard (1985) built it into a simulation model based on 
computer programming ideas, which laid a foundation for its wide application. Subsequently, CIPP model has 
been widely applied in education focusing on the evaluation of school programs, school policies, curriculum 
implementation and so on. 

In the aspect of school projects evaluation, Jumari and Suwandi (2020) described to the evaluation of 
the CIPP model on early childhood friendly programs and argued that the results of the CIPP evaluation are 
expected to contribute to Islamic educational institutions as best practices for implementing child-friendly 
school programs. Prasetiyo et al. (2020) found that the application of the CIPP model to an environmentally 
friendly program is a fully inclusive school. The results showed that the program was successful in disseminating 
motivation and environmental care to students, laying the foundation for sustainable development. Agustina 
and Mukhtaruddin (2019) described to the use of CIPP model for evaluation in Integrated English Learning 
(IEL) program, provided useful ideas for improving the program, motivating teachers to do better and more 
in the teaching and learning process. Hurmaini (2015) conducted an evaluation of evaluation of the campus 
social practice program, made some improvements to address the shortcomings exposed by the context, input, 
process, and product evaluation. 

Meanwhile, some researchers have evaluated policies using the CIPP model, such as Jamil and Iqbal (2020) 
who evaluated the implementation of a vocational training program for women’s entrepreneurial skills, and 
the findings recommended revising curriculum content, providing a better environment, and developing 
effective teaching strategies. Edwards (2016) used the CIPP model for the integration of agricultural literacy 
development education policy while enabling change agents to support teacher participants and encourage 
them to use agriculture as a context for teaching and learning. 

In terms of evaluating learning strategies, the CIPP model was used to assess conceptual framing of learn- 
ing styles, which had an impact on institutional assessment and continuous improvement in higher education 
institutions (Chinta et al., 2016). Tokmak et al. (2013) used the CIPP model to evaluate and redesign an online 
master’s degree program. The results showed that most students were satisfied with the new version of the 
program. 

On the part of curriculum evaluation, Al-Shanawani (2019) used the CIPP model to evaluate the kinder- 
garten self-learning curriculum and suggested that all aspects of preschool children’s education should be 
studied and evaluated on an ongoing and comprehensive basis. Akpur et al. (2016) evaluation of the design 
of the English preparatory curriculum at Yildiz Technical University, the findings revealed that teachers and 
students were generally positive about the curriculum. Powell and Conrad (2015) utilized the CIPP model to 
develop, among other things, an integrated service-learning component for a university health course, and the 


410 


ASS https://doi.org/10.33225/jbse/22.21.408 


Journal of Baltic Science Education, Vol. 21, No. 3, 2022 


ISSN 1648-3898 /Print/ MULTI-INDEX AND HIERARCHICAL COMPREHENSIVE EVALUATION SYSTEM FOR TRAINING 
QUALITY OF SCIENCE AND ENGINEERING POSTGRADUATES 
ISSN 2538-7138 /Online/ (Ppp. 408-427) 


results indicated that students felt that service-learning enhanced health promotion knowledge, reinforced 
topics, and increased student ownership of the course. As seen from the current achievement, the CIPP model 
has been widely used in the field of education and achieved good results in teaching evaluation. Therefore, it 
is important to build a quality evaluation system for postgraduate training in science and engineering based 
on CIPP model. 


Evaluation System Construction 


In the evaluation of training quality of science and engineering, postgraduates based on CIPP model, 
students’ enrollment basis, training objectives and training mechanism are part of the training background. 
It is a part of context evaluation that can be carried out at the beginning of semester. The input of teacher 
resources, external resources and policy support of universities in cultivation, are the basic resources and con- 
ditions of teaching implementation. It is a part of input evaluation that can be carried out at the beginning of 
semester. The types of courses, the richness of activities and the specific implementation are the main parts of 
teaching process, parts of process evaluation, which can be carried out once a month. The basic requirements, 
academic ability and social ability of the students are three parts of product evaluation. It can be carried out 
in the mid-term and the final term because of the parts of product evaluation from which we can observe that 
the evaluation indexes of science and engineering postgraduate quality training are consistent with the four 
elements of CIPP model, which can be used to construct the evaluation system. Figure 1 shows the flow chart 
of establishing the weight of engineering postgraduate quality index based on CIPP model. 

First, the main factors related to the objectives were extracted from the teaching activities according 
to the requirements of the engineering postgraduate training program and CIPP evaluation model, and the 
first-level, second-level and third-level indicators were established. The weights of each first-level and second- 
level index are calculated by using Analytic Hierarchy Process. Next, as a result of the third-level indicators, a 
questionnaire was developed based on expert opinions and training program requirements, and 756 post- 
graduates were collected for empirical analysis. After collecting the data, Factor Analysis was carried out in 
order to isolate common factors to assess the validity of the content of the second and third level indicators, 
as well as the weights of the second-level indicators. Based on the extraction of the common factors, the Bx 
indicator weights and the scores of the student samples on the common factors were calculated. The Bx index 
weights and the subjective weights of the Bx indexes are used to determine comprehensive weights of the Bx 
indexes, after which the formula for calculating the scores using the CIPP evaluation model is derived from the 
Ax index weights and the comprehensive weights of the Bx indexes in order to construct the final evaluation 
system for postgraduates in science and engineering. Finally, based on this, Cluster Analysis was used to clas- 
sify the quality of students’ training based on their scores on second-level indicators in order to understand 
the differences between students and their performance on each influencing factor. 

In this model, a total of three analysis methods were selected, namely, Analytic Hierarchy Process, Factor 
Analysis, and Cluster Analysis. Among them, the Analytic Hierarchy Process includes four steps to calculate 
the weight values of each first-level and second-level indicator (Saaty, 1993). Firstly, constructing the structure 
of the hierarchical order of the indicator system, it was divided into three levels, namely, the target level, the 
criterion level, and the implementation level. Secondly, the expert discussion method was used to determine 
the judgment matrix as well as the relative importance of each issue using the Delphi method. Thirdly, the 
relative weights of each layer were calculated, and the judgment matrix was listed according to the importance 
level of 1-9 (Seal, 1964). 

In order to detect whether the selected indicators are suitable for Factor Analysis, KMO (Kaiser Meyer 
Olkin) and Bartlett's tests were performed using SPSS (Statistical Package for the Social Sciences) 23.0 software, 
and usually a KMO value greater than 0.5 can be considered for Factor Analysis. 

Cluster Analysis includes hierarchical clustering, two-step analysis, and K-means clustering Jang & Hitch- 
cock, 2021). Since we were not sure how many specific groups to classify in this study, we chose the scheme of 
using a combination of systematic clustering and K-means clustering, firstly using systematic Cluster Analysis to 
determine the number of groups, and then using K-means clustering to obtain specific grouping information. 
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Figure 1 


Process of Establishing the Quality Index Weights for Science and Engineering Postgraduates 
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Based on the above research, this current study constructed an evaluation index system and determined its 
weights from four aspects: Context, Input, Process and Product, based on CIPP model and combined with the train- 
ing mode of science and engineering postgraduates, and designed a strategy to evaluate the quality of science 
and engineering postgraduates’ cultivation from both qualitative and quantitative analysis. Therefore, this study 
aimed to address the following questions: 

1. Is it reasonable to determine the multi-index and hierarchical comprehensive evaluation system for 
postgraduate training in science and engineering based on the CIPP model? And does each weight 
need to be adjusted in some way through the results of empirical analysis? 

2. Is there any possibility to construct a quantitative evaluation method for the quality training of science 
and engineering postgraduate based on the CIPP model? 


Research Methodology 
General Background 


Analytic Hierarchy Process is a decision-making method that decomposes the elements always related to 
decision-making into levels such as objectives, criteria, and options, on the basis of which qualitative and quantita- 
tive analysis is performed. Its main feature is the hierarchy of the decision-making process according to the indi- 
vidual’s thinking and mental patterns, which allows to summarize multi-factor problems or complex multi-criteria 
problems into a hierarchical structure. Saaty (1993) pointed out that the hierarchy is a representative structure of 
complex problems in a multi-level hierarchical structure. 

Factor Analysis refers to the study of statistical techniques that extract common factors from a population 
of variables. Factor Analysis can identify hidden representative factors among many variables. Grouping variables 
with the same essence into one factor reduces the numbers of variables and also tests the hypothesis that variables 
are related (Spearman, 1904). 

Cluster Analysis refers to the analytical process of grouping a collection of physical or abstract objects into 
multiple classes consisting of similar objects Jang & Hitchcock, 2021). It is an important human behavior. The goal 
of Cluster Analysis is to collect data to classify on the basis of similarity. 


Participants 


756 postgraduates with a master’s degree in physics and engineering were randomly selected as the objects 
via the internet, including five majors in optics, optical engineering, theoretical physics, condensed matter, and 
electronic engineering, who volunteered to participate. 
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Design of Evaluation Indicators 


Based on CIPP Model, context evaluation of science and engineering postgraduates takes postgraduate training 
plan as the core, including three second-level indexes: enrollment basis, cultivation objectives and training mecha- 
nism. Among them, enrollment basis is the assessment of students’ professional knowledge mastery, cultivation 
objectives and mechanism are the assessment of postgraduate training direction, and the latter two indicators are 
more closely related to the future development of postgraduates. Input evaluation takes educational resources as 
the core, including three second-level indexes: teacher input, external resources, and organizational guarantee. 
Among them, the teacher input index is to explore students’ satisfaction with the qualification and arrangement of 
teachers in their schools, and the external resources and organizational guarantee index is to investigate students’ 
satisfaction with the research conditions and research funding support, which is more important because teachers 
are directly involved in students’ learning. Process evaluation takes teaching process as the core, including three 
second-level indexes, namely multi courses, the richness of teaching activities and practical process. The diversity 
of courses and teaching activities is the basis for the multifaceted cultivation of students, and the practical process 
indicators are the evaluation of students’ commitment to learning, and these three indicators are of similar impor- 
tance. Product evaluation takes the postgraduate training results as the core, including the basic quality, academic 
ability, and social ability. Among them, basic quality indicators mainly evaluate students’academic completion and 
thesis writing ability, which are the most important compared with the other two indicators. 

Accordingly, the quality training evaluation index of science and engineering postgraduates based on CIPP 
model was designed, and the specific content is shown in Table 1. Using A,-A, to denote 4 first-level indicators, B,- 
B,, to denote 12 second-level indicators, and C,-C,, to represent 36 third-level indicators. Among these first-level 
indexes, the content of context evaluation is mainly the basic environment, allocation of educational resources 
belongs to input evaluation, the main content of process evaluation is the teaching and learning process, and 
academic achievement belongs to product evaluation. Using the Analytic Hierarchy Process, experts with many 
years of rich teaching management experience were invited to construct a judgment matrix, the characteristic roots 
and weights of the matrix were calculated by MATLAB software, on which the consistency test was conducted. The 
weights of context, input, process, and product evaluation in the first-level indicators were obtained as 0.11, 0.17, 
0.34 and 0.38, respectively, which means that if the total evaluation score is 100, the scores of these four evaluation 
elements are 11, 17,34 and 38, respectively. The weights are determined in the Appendix 1. It is concluded that the 
weights of the four indexes in the first-level index were 0.106, 0.167, 0.345, 0.381 and the weight of three indicators 
in the second-level index for context evaluation were 0.163, 0.296, 0.540. The weight of three indexes for input 
evaluation were 0.413, 0.327 and 0.260. The weight of three indexes for process evaluation were 0.333, 0.333 and 
0.333 and product evaluation weights of three indexes were 0.327, 0.413 and 0.260, respectively. Since there are 
too many third-level indicators, they don’t all have independent weights, and each indicator has the same weight. 


Table 1 
Evaluation Index and Weight of Engineering Postgraduate Quality Training Based on CIPP Model 


First-level index Second-level index Third-level index (Measurable quota) 


Postgraduate entrance score of specialized courses(C1) 
Postgraduate entrance interview score(C2) 

Cross-disciplinary situation of students(C3) 

The training of postgraduate scientific research ability(C4) 

The cultivation of postgraduates’ criticism and innovation ability(C5) 
Interdisciplinary postgraduate training(C6) 

The construction of industry-university-research collaborative innova- 
tion mechanism(C7) 

The construction of cooperative training mechanism in colleges and 
universities(C8) 


The training of interdisciplinary cross-integration(C9) 


Enrollment basis(B1) 
(0.163) 


Cultivation objective(B2) 
Context evaluation(A‘) (0.296) 


(0.106) 


Training mechanism(B3) 
(0.540) 
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Input evaluation(A2) 
(0.167) 


Second-level index 
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Third-level index (Measurable quota) 


On-campus supervisor qualification(C 10) 
Enterprise supervisor qualification(C 11) 


Qualifications of industry peripheral supervisor (C12) 


External resources(B5) 
(0.327) 


Factory in related fields and richness of enterprise resources(C13) 
Laboratory instrument configuration(C14) 


Richness of library resources(C15) 


Organizational guarantee(B6) 
(0.260) 


The funding of school education(C16) 
The reward system of postgraduate learning outcome(C17) 
The administrative system of postgraduate(C 18) 


Process evaluation(A3) 
(0.345) 


Multi courses(B7) 
(0.333) 


Basic professional knowledge in this field(C 19) 

Ability to use foreign languages, mathematics and computer 
software(C20) 

Social humanities courses, ideological and political courses(C21) 


The richness of teaching activities(B8) 
(0.333) 


Frontier development curriculum(C22) 
Domestic and foreign academic exchanges(C23) 
Practice design comprehensive training(C24) 


Practice process(B9) 
(0.333) 


The degree of teacher-student interaction(C25) 
The use of teaching methods and strategies by teachers(C26) 


Postgraduate classroom participation(C27) 


Product evaluation(A4) 
(0.381) 


Basic quality(B10) 
(0.327) 


Academic integrity(C28) 
Grade Point Average (GPA) score(C29) 
Postgraduate dissertation grades(C30) 


Academic ability (B11) 
(0.413) 


Postgraduate publications(C31) 
Postgraduate patent applications(C32) 
Honors and awards received at school(C33) 


Social ability(B12) 
(0.260) 


Participate in academic conferences and have results(C34) 
Participate in other social competitions(C35) 


Employ and set up a business(C36) 


Instrument 


A questionnaire about the basic information for postgraduates to evaluate the quality of postgraduate training 
in science and engineering (Appendix 2). On the basis of the above evaluation indicators, the questionnaire was 
prepared according to the content of the third-level indexes, which includes 36 items for evaluating students on 
context, input, process, and product evaluation within the whole questionnaire, including multiple-choice questions 
that set five options A-E, corresponding to 4-0 points and whether questions. The quality of postgraduate train- 
ing was evaluated according to the high score, taking the product evaluation as an example, as shown in Table 2. 


Table 2 


Take the Product Evaluation in the Questionnaire as an example 


Participate in academic conferences and have results(C,,,) 


Product evaluation——Academic achievements(A,) 


Third-level index (Measurable quota) 


Comply with academic integrity(C,,) 


GPA above 3.5(C,.) 


Participate in other social competitions(C,,) 


Employ and set up a business(C,,) 


A 


Options 
A(1 point) —_B (0 point) 
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Product evaluation——Academic achievements(A,) 


Postgraduate dissertation 


Above 90 80-89 70-79 60-69 Below 60 
grades(C,,,) 
Postgraduate papers SCI SCI (Q3, Q4) 
publications(C,,.) (at, a2) EI Core Journals General Journal Not have 
Honors and awards received at Three or more pecan and aia Third academic 
Two awards academic scholar- : Not have 
school(C,,) awards ship scholarship 
Utility model 
eeu ; patent and ap- software copyright Invention pat- 
Postgraduate patent applications(C,,) Invention patent granted pearance patent Seeined ent applied Not have 
granted 


The product evaluation for postgraduates consists of a total of nine questions, divided into five whether 
questions and four multiple-choice questions. Among the yes/no questions, there are five dimensions: whether 
the student has finished the dissertation, whether the GPA of the student is higher than 3.5, whether the student 
has participated and succeeded in academic conferences, whether the student has participated in other social 
competitions, and whether the student is employed, which are tests of the basic ability of the student's study. In 
addition, the four multiple-choice questions examine the students’ comprehensive quality ability from the score of 
the dissertation, the publication of the dissertation, the award and honor of the university, and the patent applica- 
tion. Among them, for Chinese postgraduates, publishing SCI, El and Chinese core papers as well as applying for 
invention patents, utility model patent and appearance patent, are significant achievements during the students’ 
study, which is a comprehensive evaluation of the students’ academic and comprehensive abilities. 


Reliability and Validity 


The collected data were entered into SPSS 23.0 software, the information was normalized into dimensionless 
data, and the reliability and validity of the questionnaire were verified using KMO statistical test and Bartlett's test. 
In addition, the significance level of the Bartlett’s test results was set at p = .05. The results of reliability analysis 
are shown in Table 3. 


Table 3 
Reliability Statistics of Questionnaire 


KMO and Bartlett’s Test 
Context Input Process Product 
KMO .906 .925 919 765 
Bartlett’s Test p <.05 <.05 <.05 <.05 


As seen in Table 3, the KMO values of context evaluation, input evaluation, process evaluation, and product 
evaluation were .906, .925, .919, and .765, respectively. These analyses and calculations indicated that all items are 
in positions above .500 and sig values p < .05, which means there is a strong correlation between indicators and 
Factor Analysis can be conducted. 
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Research Results 
Rationality of Index Design 


Factor Analysis was conducted through the experimental data to obtain the variance contribution rate of each 
factor to the final results. The 12 common factors selected for subsequent analysis, which had a variance contribu- 
tion greater than 85% or an eigenvalue greater than (or close to) 1, are denoted by F,-F,,. The variance contribution 
rate of Postgraduates’ Academic Achievement Evaluation is shown in Table 4. It showed that eigenvalues of the first 
two factors were greater than 1, and the eigenvalues of the third factor were approximately 1, which indicated that 
each part had three factors that could well describe the specific meaning of the corresponding nine indicators. 
These factors can be regarded as common factors corresponding to the number of second-level indexes, indicating 
that the setting of third-level indicators is reasonable. 


Table 4 
Eigenvalue and Variance Contribution Rate of Postgraduates’ Academic Achievement (Product) Evaluation 


Eigenvalues of matrix Percentage of variance (%) Cumulative contribution value (%) 
Context evaluation 
Fi 6.544 72.706 72.706 
F; 0.781 8.673 81.379 
F, 0.571 6.341 87.720 
Input evaluation 
Ey 6.724 74.708 74.708 
Fe 0.594 6.597 81.305 
F 0.392 4.352 85.658 


Process evaluation 


Fy 6.283 69.811 69.811 

F, 0.662 7.358 77.169 

F, 0.477 5.305 82.474 
Product evaluation 

Fig 2.921 32.453 32.453 

Ee 1.280 14.222 46.675 

F 0.995 11.052 57.727 


According to rotation load matrix in the Factor Analysis, the influence of each variable on each factor can be 
analyzed. The rotated load matrix is the factor load matrix obtained by deriving a deterministic solution from the 
uncertain solution that gives a meaningful interpretation of the common factors and transforming the initial factor 
load matrix according to the simple structure criterion. The specific coefficients are shown in Table 5. On the basis 
of coefficient of each factor on the common factor, the influence degree of each factor on the common factor was 
judged. In light of the requirements of Factor Analysis, the closer the KMO value is to 1, the more suitable for Factor 
Analysis, more than 0.8 is also more suitable, 0.6 is more general, and below 0.5 is not suitable for Factor Analysis, so 
the factors with absolute values of coefficients greater than 0.5 are selected and the common factors are extracted and 
compared with evaluation content of the second-level index to test rationality of the design of the second-level index. 

Taking Factor Analysis rotation load matrix of the product evaluation as an example, as can be seen that the 
coefficients of common factor F,, in” paper publication’’patent application’ and “national scholarship acquisition” 
were 0.854, 0.754 and 0.726 respectively, which mainly reflects the students’ scientific research ability, innovation 
and expansion ability, engineering technology and practice ability. Therefore, the common factor F,, was named 
as “academic ability factor”. The coefficients of common factor F,, on “academic integrity score” “GPA score” and 


A416 


ASS https://doi.org/10.33225/jbse/22.21.408 


mu 


Journal of Baltic Science Education, Vol. 21, No. 3, 2022 


ISSN 1648-3898 /Print/ MULTI-INDEX AND HIERARCHICAL COMPREHENSIVE EVALUATION SYSTEM FOR TRAINING 
QUALITY OF SCIENCE AND ENGINEERING POSTGRADUATES 
ISSN 2538-71 38 /Online/ (Ppp. 408-427) 


“dissertation score” were 0.591, 0.674 and 0.669, reflecting the basic abilities of postgraduates such as knowledge 
learning, literature reading and thesis writing. Accordingly, the common factor F,, was named as “basic quality 
factor” The coefficients of common factor F,, in“academic conference participation and awards’“other social com- 
petition awards” and “employment and entrepreneurship” were 0.687, 0.600 and 0.704. These awards are mainly 
based on entrepreneurship competitions, academic communication, and social activities in other fields, reflecting 
the students’ communication and presentation skills, team cooperation ability and the ability to act. Therefore, the 
common factor F,, was named as “social ability factor”. As a result, it shows that the name design of three second- 


level indexes is in line with the data analysis, and the design is more reasonable. 


Table 5 
Factor Analysis of Postgraduate Academic Achievement (Product) Evaluation Rotating Load Matrix 


Component 
Third-level index 
EF, E, Fs 
Comply with academic integrity(C,,,) 0.150 0.591 0.264 
GPA above 3.5(C,,) 0.530 0.674 0. 061 
Results of postgraduate dissertation(C,,,) 0.605 0.669 -0.002 
Situation of papers published(C,,,) 0.854 0.114 0.061 
Situation of patent applications(C,,,) 0.754 -0.116 -0.189 
Honors and awards received at School(C,,) 0.726 -0.080 0.078 
Participate in academic conferences and have results(C,,,) 0.344 0.176 -0.687 
Participate in other social competitions(C,,) -0.233 0.374 0.600 
Employ and set up a business(C,,,) 0.272 0.254 0.704 


In the evaluation system of quality training of postgraduates in science and engineering based on the CIPP 
model, the second-level indexes were designed to correspond to three factors, and based on this, the third-level 
indicators that can be easily measured were prepared. Through the Analytic Hierarchy Process, the weights of the 
second-level indexes were established in the initial establishment of the evaluation system. On this basis, the scores 
of the experimental samples on each third-level indexes were collected through the questionnaire method, and 
the common factors affecting the scores of these third-level indexes were identified by using Factor Analysis. The 
coefficient and naming of third-level index on corresponding common factor are summarized in Table 6. 


Table 6 
Coefficient and Naming of Third-Level Index on Corresponding Common Factor 


Third-level index Common factor 
C, C, C, 
F, 0.777 0.745 0.862 Alignment of training objectives with student needs 
C, C, C, 
F, 0.562 0.823 0.877 Student's academic foundation factor 
C, C, C, 
F, 0.719 0.771 0.869 Culture mechanism factor 
Ci C,, C,, 
F, 0.635 0.718 0.843 School organization guarantee factor 
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Third-level index Common factor 
C,, C,, C., 
F, 0.724 0.583 0.774 External resources factor 
Ci, C,, Cig 
F, 0.816 0.814 0.537 Teacher resources factor 
Ci, C,. C,, 
F, 0.720 0.840 0.758 Activity richness factor 
C,, C5 C,, 
F, 0.815 0.856 0.876 Factor of curriculum diversity degree 
C,, C,, C,, 
F, 0.801 0.702 0.982 Practical factor 
C,, C,, Cy 
Fa 0.854 0.754 0.726 Academic ability factor 
C. C,. Cy, 
Re 0.591 0.674 0.669 Quality foundation factor 
C,, C,, Cy 
F 0.687 0.600 0.704 Social ability factor 


It shows that the common factors affecting the experimental sample data can be divided into four catego- 
ries based on the four elements of the CIPP model, with three factors in each category, and the specific contents 
are consistent with the preliminary construction of the second-level index indicating that the establishment of 
second-level indexes is more reasonable. For example, in terms of product evaluation, three second-level indexes 
were designed in the initial construction of second-level indexes, and through Factor Analysis, it is found that the 
common factor of “participation and awards in academic conferences,’ “awards in other social competitions,’ and 
“employment and entrepreneurship” can be summarized as “social competence factor’, which is in line with the 


previous design. 
Weight Adjustment 
The scores of the student on the 12 common factors were used as variables and again downscaled, which 
allowed the scores of each second-level index for the first-level index to be derived as the objective weights of 


the three second-level indexes. The factor weight of postgraduate quality training evaluation is shown in Table 7. 


Table 7 
Factor Weight of Postgraduate Quality Training Evaluation 


Corresponding weights of the second-level index 


B, B, B, 

Context 0.100 0.291 0.609 
B, B, B, 

Input 0.414 0.341 0.245 
B, B, B, 

Process 0.267 0.354 0.379 
B, B,, B, 

Product 0.316 0.436 0.248 
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We can see that the scores of each second-level index for the first-level index, which are the objective weights 
of the three second-level indexes, 0.316, 0.436 and 0.248, respectively. Through comparing the subjective weight 
(0.327, 0.413, 0.260) formulated by Analytic Hierarchy Process with the objective weight, it is found that the dif- 
ference was not significant, and the previous subjective judgment was more reasonable. In addition, through the 
weighing method of objective weight and subjective weight proposed by Song and Wang (2003), the comprehensive 
weights were defined as the arithmetic mean value of subjective and objective weight. 

The weight adjustment of the second-level index of the product evaluation is (0.322, 0.424, 0.254). Using 
the same method, it can be concluded that the weights of second-level indexes of context evaluation are (0.132, 
0.293, 0.575), the weights of second-level indexes of input evaluation are (0.413, 0.334, 0.253), and the weights of 
second-level indexes of process evaluation are (0.300, 0.344, 0.356). 


Quantitative Evaluation Mode and Improvement 


After determining the weight of first and second-level indexes of the evaluation system, the initial calcula- 
tion formula of student quality evaluation score was obtained, assuming that the total score is 100 points. In the 
following equation, F,, stands for the product evaluation score of the common factors. 


e=17 [ 0.132 (C,+C,+C, )+0.293(C,+C,+C, )+0.254(C,+C,+C, | 
+1 1 0413(C,40 70, 0334 ( CFC tC, JA 253(C ACG. | 
+34 0.300(Cyy tC oy tC, )+0.344 (Cop +Co5 +Cog )+0.356(Co5 tC a tC) | 
+38[ 0.322 (Cog +Co9+Cp ) +0.424 (Cy, +C5y +C,3 ) 40.254 (Cy, +Cy5+Cy,) | 
At this moment, C, needs to be in the unified range, and the use has limitations. In order to expand the scope 
of use, data need to be normalized before calculation and become dimensionless data. Therefore, the coefficient 


of each three-level index on corresponding common factor can be calculated according to existing data through 
Factor Analysis. For example, the factor score matrix of the product evaluation is shown in Table 8. 


Table 8 
Factor Score Coefficient Matrix of Postgraduate Academic Achievement (Product) Evaluation 


Component 
Third-level index 
F. EF. F., 
Comply with academic integrity(C,,,) -0.072 0.410 -0.115 
GPA above 3.5(C,,) 0.136 0.200 0.017 
Results of postgraduate dissertation(C,,,) 0.084 0.384 0.047 
Situation of papers published(C,,,) 0.329 -0.007 -0.037 
Situation of patent applications(C,,,) 0.339 -0.164 0.142 
Honors and awards received at school(C,.) 0.329 -0.153 -0.104 
Participate in academic conferences and have results(C,,,) 0.059 0.125 0.703 
Participate in other social competitions(C,,) -0.268 0.509 0.278 
Employ and set up a business(C.,,) 0.074 0.180 -0.570 


Table 8 shows linear combination relationship between common factors F, F,, and F,, and each three-level 
index, and the scores of F,,, F,, and F,, can be calculated respectively according to the expression of the factor 
score function, the score equation of first-level index is established and score coefficients of other indicators can 
also be obtained through investigation and Factor Analysis (Appendix 3). 

To sum up, a more reasonable evaluation system of science and engineering postgraduate training quality 
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based on CIPP model is established by Factor Analysis on the existing basis. When evaluating the quality of en- 
gineering postgraduate training, we can intuitively understand the situation of postgraduate quality training by 
comparing the comprehensive score. By analyzing and comparing the scores of each first-level index or second-level 
index, we can accurately understand the implementation of each measure and training results and put forward 
effective suggestions. 


Student Classification 


According to the results of Factor Analysis, the scores of each student in academic ability (F,,), basic quality 
(F,,), and social ability (F,.) are partially presented in Table 9. 


Table 9 
756 Students’ Scores on the Second-Level Index 


ae Fi Fy; 
1 -1.170 -0.669 0.874 
2 1.976 -0.075 1.324 
3 1.259 -0.258 0.696 
753 -1.185 -0.101 -0.686 
754 -1.090 -0.742 -0.784 
756 -1.640 -1.749 0.762 


Based on the scores of middle school students in F 
classification steps was obtained, as shown in Figure 2. 


F,,and F,, in Table 9, a pedigree chart describing the 


10! 


Figure 2 
Hierarchical Cluster Analysis Pedigree Chart 
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144 108 312 186 6 
Clustering combination of rescaling 


The students can be divided into five categories by observing the pedigree, and then the K-means clustering 
is used to obtain the grouping data, as shown in Table 10. 
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Table 10 


Cluster Analysis Results of 756 Students 


ae Group (Number) = Group 1(144) Group ll (108) Group lll(312) Group IV(186) Group V (6) 


F,, (Academic ability) -1.117 0.978 0.616 -0.706 -0.952 
F,, (Basic quality) -0.877 0.122 -0.094 0.989 6.930 
F,, (Social ability) -0.074 1.700 -0.644 0.088 1.979 


It can be seen from Table 10 that there are 144 students in Group |, whose abilities in all aspects are at the 
average level, and they are slightly deficient in the social ability (F,,). There are 108 students in Group II, whose 
learning degree is relatively good overall, and they have published papers in journals such as SCI. Most of them have 
patents that are being applied for or have been issued. Some students have applied for national scholarships and 
actively participated in various national conferences, with stronger abilities in all aspects. There are 312 students 
in group Ill, who had a relatively average degree of study, scored above 80 points in dissertations during their 
master’s degree studies, no journal papers published or only published in general journals, and they participated 
in fewer academic conferences and exchanges. They had mastered basic professional knowledge and scientific 
research ability. A total of 186 students in the fourth category who have published papers in journals such as El and 
Chinese core. Some students have patents being applied for or issued and have a good degree of study. There are 
6 students in Group V who had no experience of applying for patents or publishing papers during their study but 
had participated in competitions held in other fields of society, and received awards, and had performed slightly 
in terms of academic ability and basic quality. 


Discussion 
Discussion on Model Application 


In this study, according to the CIPP evaluation model and the requirements of postgraduate training in sci- 
ence and engineering, the process of constructing a quality evaluation system for postgraduate training in science 
and engineering is designed, and the rationality of indicators is tested as well as empirical analysis is conducted 
by Analytic Hierarchy Process, Factor Analysis and Cluster Analysis. The results showed the reasonableness of the 
indexes, and then designed the way to evaluate the quality of postgraduates in science and engineering. Based on 
the analysis results, effective measures to promote the development of postgraduates in science and engineering 
can be actively proposed, and the specific measures are as follows. 

Establish a high level of dual type mentor group faculty. Cultivating innovative talents with practical 
experience has become one of the basic goals of higher education talent cultivation (Qi & Peng, 2018). Postgradu- 
ates are required to have certain basic theories as well as strong practical skills and innovation ability. In Chinese 
universities, there are a large number of teachers with solid basic theories to serve as on-campus supervisors for 
postgraduates in science and engineering, but the on-campus supervisors have less practical experience. This can 
make up for the disadvantage of having less practical experience for the on-campus instructors, and the off-campus 
instructors can provide rich cases to guide the students. Colleges and universities can establish industry-university- 
research cooperation with enterprises and appoint enterprise teachers and on-campus supervisors to jointly guide 
the research and practice of postgraduates. 

Reconstruct the existing resource system of postgraduate education. Postgraduate cultivation needs to rely 
on relevant resources outside the education system and reconstruct the existing postgraduate education resource 
system (Cheng & Wang, 2010). In order to improve postgraduate training, enterprises with similar majors can be 
invited to participate in training postgraduates, combining the supervisor's subject with enterprise, performing 
innovative research in conjunction with the actual situation, cooperating with enterprises in industry-university- 
research, and establishing practice bases. In addition to providing relevant training, postgraduates should be 
encouraged to engage in work under the cultivation of cooperative enterprise units in combination with their 
supervisors’ topics. Under the guidance of professionals, enterprise project research and development can also 
be accomplished. By combining postgraduate innovation training with practical application, these resources can 
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be integrated into the postgraduate training system. It is also possible to broaden the horizons of postgraduates 
through the university's foreign exchanges and cooperation, so that they can understand the research dynamics 
and frontiers of their disciplines and improve their innovation ability. 

Improve the construction of postgraduate curriculum system. Take the cultivation objectives and degree 
requirements as the fundamental basis for the design of the curriculum system, implement the postgraduate cul- 
tivation objectives and degree requirements completely, and pay attention to the systematic design and overall 
optimization of the curriculum system (Ministry of Education, 2020b). It insists on taking ability cultivation as the 
core and innovation ability cultivation as the focus, broadening the knowledge base and cultivating humanistic 
literacy, strengthening the integration and articulation of curriculum systems in different cultivation stages, de- 
signing the curriculum scientifically. 

Enhance the ability of postgraduates to conduct scientific research. Throughout the three years of post- 
graduate training, basic research training is provided to postgraduates in order to enhance their research abilities 
(Wu et al., 2019). As a result, on the one hand, all the training time can be utilized, while on the other hand, students’ 
research abilities can be strengthened. On this basis, the project serves as a guide to enable postgraduates to integrate 
into research activities as soon as possible and maximize the impact of cultivating postgraduates’ research abilities. 


Discussion on Student Classification Using K-means Clustering 


In this study, the students were divided into five groups by first using systematic Cluster Analysis to obtain a 
lineage diagram describing the classification steps, and then using the K-means clustering to obtain specific group- 
ing information. The results of the Cluster Analysis showed that the students in group II had the best academic 
ability, basic quality, and social ability among the five groups. 

As seen that the mechanism of industry-university-research collaboration has begun to take shape and 
achieve results in the cultivation of science and engineering in Chinese universities (Yang & Li, 2012). About 60% 
of students have access to more advanced hardware equipment in this learning environment, have a wide range 
of opportunity for communication and selection, and can be more clear about the combination of experimental 
purposes and social needs. At the same time, factories and enterprises can help students to clarify scientific re- 
search objectives and find the combination of their own needs and social needs, which improves the applicability 
of research results. It is conducive to the improvement of postgraduates’ interest in scientific research, scientific 
inquiry ability and hands-on practical ability (Shen & Zhang, 2015). 

About 50% of the students have applied for patents and published corresponding papers, among which 
only 20% of them are the first applicants, and the remaining students mostly apply with their supervisors or off- 
campus supervisors, which shows that the dual tutorial system has played its role in promoting the development 
of students’ innovative ability and increasing the diversity of supervisors’ guidance. At the same time, off-campus 
supervisors, such as some high-level engineers and experts, can explain emerging technologies and methods used 
in practical production for students, guide students to carry out research about the social production practice and 
work needs, and stimulate postgraduates’ thinking and innovation through practice. 

Nearly 65% of the students have attended more than three academic conferences with their supervsors or 
project teams, and the university also frequently invites experts in relevant disciplines to hold academic lectures and 
exchange events. They create a strong academic atmosphere for students, keep them abreast of the latest scientific 
research achievements, and expand their academic horizons. This helps students to grasp professional knowledge 
at the macro level and further study issues of interest to them. However, there are still some students who seldom 
participate in academic exchange activities because of their courses, experiments, or social activities. Therefore, 
schools should establish a rich online learning platform to bring students closer to academic communication. For 
example, public teaching platforms, learning resource forums and MOOC courses. 

About 60.3% of students’ social skills needs to be improved, and the poor integration of academic research and 
social communication activities is one of the important reasons for this unbalanced development (Barton-Arwood 
etal., 2005). Therefore, while increasing social exchange activities, schools should integrate academic research into 
them, such as holding university exchange meetings, providing opportunities for students to communicate with 
different schools and students of different majors, strengthening cooperation between colleges and departments, 
and sharing teaching resources. On the other hand, it motivates students to obtain help from researchers in related 
fields and peer learning possesses a stronger incentive effect, and social skills such as oral and written expressions 
have also been cultivated on the other. 


422 


Ass https://doi.org/10.33225/jbse/22.21.408 


Journal of Baltic Science Education, Vol. 21, No. 3, 2022 


ISSN 1648-3898 /Print/ MULTI-INDEX AND HIERARCHICAL COMPREHENSIVE EVALUATION SYSTEM FOR TRAINING 
QUALITY OF SCIENCE AND ENGINEERING POSTGRADUATES 
ISSN 2538-7138 /Online/ (Ppp. 408-427) 


Conclusions and Implications 


In China, with the continuous expansion of the enrollment scale of postgraduates, the mode of postgraduate 
training and the quality of training have become more and more widely concerned by the society. However, there 
are problems in the process of postgraduate training in China, such as unreasonable distribution mechanism and 
unsound management and evaluation system, so there is a real need for reform of the postgraduate training system. 
In this study, based on the four elements of CIPP model, the indexes for evaluating the quality of postgraduate 
training in engineering are designed, and the weights of the first and second level indexes are calculated by using 
Analytic Hierarchy Process. Secondly, the rationality of the content and weight design of the second-level index 
corresponding to the product evaluation is proved through empirical analysis, and the weights of the second-level 
index are adjusted. Finally, based on the results of Factor Analysis, the quantitative evaluation of the quality of 
engineering postgraduate training was designed. The index system can basically reflect a variety of factors affect- 
ing the quality of postgraduate training in science and engineering, which can provide some reference for each 
postgraduate training and play a realistic role in the development of scientific research abilities and comprehensive 
abilities of postgraduate students in science and engineering. 

However, compared with other disciplines, engineering postgraduates focus more attention on the applica- 
tion of scientific research results, the development of scientific and technological productivity and serving the 
development of modern factory enterprises, namely, they have to meet both the external demands such as the 
development of the main industry and scientific and technological innovation, and the internal demands of the 
development of students’ academic and social abilities. With such complex training needs, evaluators need to 
design a more comprehensive and precise evaluation scheme. The weights of the second-level index given in the 
paper are influenced both by the subjective perceptions of the evaluator and the selected research sample, and 
their coefficients are not directly applicable to the evaluation of other schools. In practice, the objective weights 
need to be recalculated based on the actual situation of the school and combined with the subjective weights to 
arrive at the final second-level index weights. At the same time, it is necessary to collect information about the 
experimental samples to verify the degree of matching between the designed second-level and third-level indica- 
tors and the actual situation of the experimental samples to make the evaluation more accurate. 
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Appendix 1 
Determination of weights using Analytic Hierarchy Process 


Firstly, the complex problem is decomposed into different elements, and then the relative importance of dif- 
ferent elements in the same level is determined by two-by-two comparison, and then the corresponding matrix 
is listed according to the 1-9 importance scales, and the relative importance weights of elements in each level are 
obtained after calculation and random consistency test. is assumed to be the importance score of indicator over 
indicator , and the judgment matrix is listed through a questionnaire survey of several experts: 


1 1/2 1/3 1/3 
Wasdes: L~ Ay2t/s 

Boe a ap 

ae eee 


where and, the eigenvectors d= [0.106879, 0.167274, 0.344545, 0.381302] and the maximum eigenvalue A 
of the judgment matrix are solved using the sum-product method, and the eigenvectors are the corresponding 
weights of each index, and then the consistency difference of the judgment matrix A is tested to see if it is within 
the acceptable range. 


(Anes =) 
CI.= G1) (1) 
CR =o (2) 


According to the formula (1) and (2), the C.R. of the judgment matrix A = 0.017 < 0.1, which means that the 
matrix is more consistent, and after rounding the obtained data di to two decimal places, the weights of the con- 
text, input, process and product evaluation in the first-level index are derived, which are 0.11, 0.17, 0.34 and 0.38, 
respectively, that is, if the total evaluation score is 100, then the four evaluation elements are 11, 17, 34 and 38 
respectively. The same method is used to calculate the weights of the second-level indicators Bx. 
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Appendix 2 

Items 


1. Postgraduate entrance score of specialized courses 

2. Postgraduate entrance interview score 

3. Interdisciplinary situation of postgraduates 

4. Satisfaction with the school policy on the development of postgraduate scientific research skills 

5. Satisfaction with the school policy on the cultivation of critical and creative skills of postgraduates 

6. Satisfaction with the school policy on the development of interdisciplinary postgraduates 

7. Satisfaction with the construction of industry-university-research collaborative innovation mechanism 


8. Satisfaction with the construction of cooperative training mechanism in colleges and universities 


9. Satisfaction with the interdisciplinary integration training 

10. Satisfaction with the qualifications of tutors in their schools 

11. Satisfaction with the qualifications of business tutors in their schools 

12. Satisfaction with qualifications of external mentors in the school sector 

13. Satisfaction with the richness of factories and enterprise’ resources in relevant fields in the school 
14. Satisfaction with the configuration of laboratory instruments in the school 

15. Satisfaction with the richness of the school library resources 

16. Satisfaction with funding for school education 

17. Satisfaction with the reward system for academic achievements of school postgraduates 


18. Satisfaction with the postgraduate administrative system 


19. Satisfaction with the cultivation of basic professional knowledge in this field 

20. satisfaction with the cultivation of foreign language, mathematics, and computer software skills 
21. Satisfaction with the social and humanities and ideological and political courses 

22. Satisfaction with the curriculum for cutting-edge development 

23. Satisfaction with the setting of academic exchanges at home and abroad 

24. Satisfaction with integrated training settings in practice design 

25. Satisfaction with teacher-student interaction 


26. Satisfaction with teachers’ teaching methods and strategies 


27. Satisfaction with postgraduates’ classroom participation 

28. Whether the academic integrity complied or not 

29. Whether the GPA (Grade Point Average) score above 3.5 or not 

30. Postgraduate degree dissertation scores 

31. Postgraduates publications 

32. Postgraduates patent applications 

33. Postgraduates honors 

34. Whether participated in the academic conference or not 

35. Whether participated in other social competitions and awards or not 


36. Whether employed and set up a business or not 
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Appendix 3 


Take the product evaluation as an example and calculate the score of the common factor 


Table 8 shows linear combination relationship between common factors F, F,, and F,, and each three-level 
index, the score of the common factor can be calculated based on the following expression of the factor score 
function: 


36 
Fm =), 1 D, gC; 
In the product evaluation, m is 10-11, Diis the score coefficient of Ci, and Ciis the measurable index, ps means 
product evaluation score. 


F,op,= -0.072C 9¢+0.136C 35 +0.084C 59 -0.329C,, +0.339C,, +0.329C,, +0.059C,,-0.268C,, +0.074C,, 
F,,,= 0.410C,,+0.200C,, +0.384C,,-0.007C,,-0.164C,,-0.153C,,+0.125C,,+0.509C,,+ 0.180C,, 


Fip,= -0.072C y¢+0.136C 35 +0.084C yg -0.329C,, +0.339C,, +0.329C,, +0.059C,,-0.268C,; +0.074C 4, 


The score equation of first-level index is established. Assuming a total score of 100, Fs represents the score of 
common factors. Weight adjustment of the second-level index of the product evaluation is (0.322, 0.424, 0.254). 
The total academic achievement is confirmed by 


F.= 100x0.38(0.322F,,,, +0.424F,,,.+0.254F,, ) 


10ps 1 lps 12ps 


Score coefficients of other indicators can also be obtained through investigation and Factor Analysis. Among 
these, F_ means comprehensive score of the common factors F,, represents foundation score of the common factors, 
F .stands for resources score of the common factors F,, means teaching score of the common factors, F.. represents 
product evaluation score of the common factors. The final evaluation score is as follows: 


F= 11a, FU s Bsa ek E38 sf, 
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